21  Scientific misconduct and the reproducibility crisis

Learning Objectives

After completing this tutorial you should be able to

Download the directory for this project here, make sure the directory is unzipped and move it to your bi328 directory. You can open the Rproj for this module either by double clicking on it which will launch Rstudio or by opening Rstudio and then using File > Open Project or by clicking on the Rproject icon in the top right of your program window and selecting Open Project.

There should be a file named 21_reproducibility-research.qmd in that project directory. Use that file to work through this tutorial - you will hand in your rendered (“knitted”) quarto file as your homework assignment. So, first thing in the YAML header, change the author to your name. You will use this quarto document to record your answers. Remember to use comments to annotate your code; at minimum you should have one comment per code set1 you may of course add as many comments as you need to be able to recall what you did. Similarly, take notes in the document as we discuss discussion/reflection questions but make sure that you go back and clean them up for “public consumption”.

  • 1 You should do this whether you are adding code yourself or using code from our manual, even if it isn’t commented in the manual… especially when the code is already included for you, add comments to describe how the function works/what it does as we introduce it during the participatory coding session so you can refer back to it.

  • 21.1 What even are these ‘Methods’ you speak of?

    Consider this

    Think back on scientific papers you have read or lab reports you have written and list the main sections of the typical format you encounter.

    Did it!

    [Your answer here]

    Consider this

    Give a brief description of what the “Methods” or “Method & Materials” section of a scientific paper or report contains and assess whether or not you think this section generally meets the standard you have just laid out.

    Did it!

    [Your answer here]

    One framework to categorize components of the “methods” of a study is to place them into three steps that build on each other.

    1. Acquire data
    2. Process data
    3. Analyze data
    Consider this

    Use the example of our recent (re)analysis of the Shark Nursery study or the fungi diversity study as an example to describe what each of these steps entails.

    Did it!

    [Your answer here]

    21.2 Tools of the trade

    Tiny history lesson Lotus123 is a what launched IBM Personal Computers into offices around the world …

    Lotus123

    My very first PC adventures included running Lotus123 off of big floppy disks. Microsoft developed Excel and the Office package and quickly PCs where not only in offices and homes around the world. Spreadsheet applications were initially focused mainly on managing and organizing data (think HR department and payroll) but increasingly complicated calculations were possible and soon Excel sneakily made its way into scientific research.

    Today, the tools used by scientists to analyze their data vary from highly specialized tools for very specific tasks (each with their own required data format), to large software packages like SAS and STATA. Many of these tools have GUIs (graphic user interfaces) and are what is frequently referred to as “point & click” or “WYSIWYG”2.

  • 2 WYSISWG = What you see is what you get. Compare this to WYWIWYG = what you want is want you get.

  • Consider this

    Rely on your experience of having used programs like Excel, SAS/STATA, or other specialized programs in a statistics class or other lab courses you have taken to compare and contrast using those types of programs for data analysis to programing languages like R, Perl or Python. Discuss the benefits and drawbacks of both approaches.

    Did it!

    [Your answer here]

    21.3 Impacts of Flawed Data Analysis

    21.3.1 Case Study 1

    Consider this

    Pull up this paper on the connection of CSF interleukin-6 and depression. Quickly skim the abstract and summarize the main result. Do this before untoggling the solution button for part 2 of this question.

    Did it!

    [Your answer here]

    Okay, fine - that was a little bit of a trick question; did you notice the retraction notice?

    You can access a more detailed version of the retraction here. Read the retraction notice and briefly describe why the paper needed to be retracted.

    After the authors fixed their mistake they published a corrected version of the paper. Compare the titles & abstracts of the retracted and updated papers to determine if their mistakes had a significant impact on the reported results.

    Consider this

    Give a brief summary of key lessons you have learned from the case study.

    21.3.2 Case Study 2

    Before today’s class you should have completed these readings:

    Consider this

    Briefly discuss why the paper needed to be retracted; as part of your discussion compare and contrast the correspondence published by Shariff et al. to the retraction notice from the original authors Consider the following (and of course add anything else you noticed/think is important).

    • Why was the paper retracted?
    • What is the central statement being made in the correspondence/retraction notice?
    • What is the tone of the correspondence/retraction notice?
    • How could this mistake have been avoided?
    • How could bias of both sets of authors/previous knowledge have played into the conclusions of the paper/noticing and identifying the mistakes (consider both sets of authors)?
    Did it!

    [Your answer here]

    Consider this

    The results of the initial study were widely published. By contrast, the retraction was not; matter of fact most people “doing their own research” on the interwebs are considerably more likely to come across an article describing the initial findings and we all know from personal experience how infrequently we look up the original study being described - so chances are they might never discover that the study has been retracted.

    Use this specific example to illustrate what Brandolini’s Law (i.e. the Bullshit Asymmetry Principle) is.

    Brandolini’s Law
    Did it!

    [Your answer here]

    Consider this

    Give a brief summary of key lessons you have learned from this case study.

    Did it!

    [Your answer here]

    21.3.3 Case Study 3

    Before today’s class you should have read key sections from Herndon et al. 2014 “Does high public debt consistently stifle economic growth? A critique of Reinhart & Rogoff”:

    • Section 1 (Introduction)
    • Section 2 (Public impact and policy relevance)
    • Introduction to section 3. (Replication) + Section Headers
    • Section 4 (Conclusion)
    Consider this

    Briefly summarize the major points of criticism of the original paper.

    Did it!

    [Your answer here]

    Consider this

    Briefly discuss the tools that were used for analysis and determine how they could have contributed to the flawed analysis being published (remember, it did get through peer-review!)

    Did it!

    [Your answer here]

    Consider this

    Recall our initial classification scientific misconduct along a continuum of “Responsible - Ethical - Unethical/Misconduct” and use that to compare the flawed analysis laid out here to the issues that were uncovered in the previous two case studies. Consider factors including content, when/why mistakes were uncovered, and intent.

    Did it!

    [Your answer here]

    Consider this

    Consider the policy and decision-making implications of this (and the other two papers) and discuss if you think retracting the actual paper with mistakes in it is sufficient.

    Did it!

    [Your answer here]

    Consider this

    Propose strategies that could be used to avoid flawed analysis demonstrated in this (and the other) case studies to be published in the first place and whose role it should be to “enforce” responsible conduct.

    Did it!

    [Your answer here]

    Consider this

    Give a brief summary of key lessons you have learned from this case study.

    Did it!

    [Your answer here]